DPL: default to transport=shmem on the same host by ktf · Pull Request #3591 · AliceO2Group/AliceO2

ktf · 2020-05-15T09:05:03Z

No description provided.

ktf · 2020-05-15T09:05:22Z

davidrohr · 2020-05-15T09:06:28Z

@ktf : Please wait a bit merging this. I am seeing some issues in the TPC workflow, when I use this in multiple places. Is there perhaps a limit on the total size of shared memory buffers I can use?

ktf · 2020-05-15T09:20:12Z

I think the default by FairMQ is 2GB.

ktf · 2020-05-15T09:20:49Z

you can check in the device log, actually.

davidrohr · 2020-05-15T09:46:15Z

I assume you mean this: shm-segment-size = 2000000000 which indeed defaults to 2 GB.

I am certainly exceeding the 2GB in my test, but I would have expected some error message during allocation. Instead it crashes. I'll check why and where and then report back.

Another note: I tried to increase the size using the --shm-segment-size 4000000000 option. I assume this defines a global shared memory for the whole workflow, or is it per processor?
If I have a merged workflow, for which binary do I set the size?

davidrohr · 2020-05-15T10:09:20Z

@ktf @matthiasrichter @rbx @shahor02 : I tried with all the latest versions + this PR to test with shared memory, and I see 2 problems:

When I call DataAllocator::make<...> and there is insufficient shared memory available, it goes stuck in an endless loop and will never return. Backtrace below. I don't think we want such a bahavior. If there is insufficient memory available, it should throw an error. At least on the laptop for tests.
The setting of the shared memory size with --shm-segment-size 4000000000 is weird for merged workflows. It seems to work correctly when I add the option to each workflow binary. But adding it only to one of the workflow binaries, e.g. the last ones, it works sometimes, sometimes it doesn't. Perhaps it is then kind of a race condition what is the size of the shared segment?

I think these 2 issues should be addressed before this PR is merged.

Stack trace when stuck in allocation:

#0  0x00007f7c995d8569 in nanosleep () from /lib64/libpthread.so.0
#1  0x00007f7c96bbbd79 in std::this_thread::sleep_for<long, std::ratio<1l, 1000l> > (__rtime=...) at /usr/lib/gcc/x86_64-pc-linux-gnu/9.3.0/include/g++-v9/thread:378
#2  fair::mq::shmem::Message::InitializeChunk (this=0x55c29ed9ff80, size=2147483648) at /home/qon/alice/sw/SOURCES/FairMQ/v1.4.16/v1.4.16/fairmq/shmem/Message.cxx:114
#3  0x00007f7c96cb1679 in fair::mq::shmem::Message::Message (this=0x55c29ed9ff80, manager=..., size=<optimized out>, factory=<optimized out>) at /home/qon/alice/sw/SOURCES/FairMQ/v1.4.16/v1.4.16/fairmq/shmem/Message.cxx:53
#4  0x00007f7c96cc1677 in fair::mq::tools::make_unique<fair::mq::shmem::Message, fair::mq::shmem::Manager&, unsigned long const&, fair::mq::shmem::TransportFactory*> () at /home/qon/alice/sw/SOURCES/FairMQ/v1.4.16/v1.4.16/fairmq/tools/CppSTL.h:25
#5  fair::mq::shmem::TransportFactory::CreateMessage (this=0x55c29cb12b30, size=2147483648) at /home/qon/alice/sw/SOURCES/FairMQ/v1.4.16/v1.4.16/fairmq/shmem/TransportFactory.cxx:128
#6  0x00007f7c96cf9f08 in fair::mq::ChannelResource::do_allocate (this=0x55c29cb12b58, bytes=<optimized out>) at /home/qon/alice/sw/SOURCES/FairMQ/v1.4.16/v1.4.16/fairmq/MemoryResources.cxx:20
#7  0x00007f7c9c6a5751 in boost::container::pmr::memory_resource::allocate (alignment=1, bytes=2147483648, this=<optimized out>) at /usr/include/boost/container/pmr/memory_resource.hpp:44
#8  boost::container::pmr::polymorphic_allocator<char>::allocate (this=<optimized out>, n=2147483648) at /usr/include/boost/container/pmr/polymorphic_allocator.hpp:82
#9  std::allocator_traits<boost::container::pmr::polymorphic_allocator<char> >::allocate (__n=2147483648, __a=...) at /usr/lib/gcc/x86_64-pc-linux-gnu/9.3.0/include/g++-v9/bits/alloc_traits.h:306
#10 std::_Vector_base<char, boost::container::pmr::polymorphic_allocator<char> >::_M_allocate (__n=2147483648, this=<optimized out>) at /usr/lib/gcc/x86_64-pc-linux-gnu/9.3.0/include/g++-v9/bits/stl_vector.h:343
#11 std::_Vector_base<char, boost::container::pmr::polymorphic_allocator<char> >::_M_create_storage (__n=2147483648, this=<optimized out>) at /usr/lib/gcc/x86_64-pc-linux-gnu/9.3.0/include/g++-v9/bits/stl_vector.h:358
#12 std::_Vector_base<char, boost::container::pmr::polymorphic_allocator<char> >::_Vector_base (__a=..., __n=2147483648, this=<optimized out>) at /usr/lib/gcc/x86_64-pc-linux-gnu/9.3.0/include/g++-v9/bits/stl_vector.h:302
#13 std::vector<char, boost::container::pmr::polymorphic_allocator<char> >::vector (__a=..., __n=2147483648, this=<optimized out>) at /usr/lib/gcc/x86_64-pc-linux-gnu/9.3.0/include/g++-v9/bits/stl_vector.h:508
#14 o2::framework::MessageContext::ContainerRefObject<std::vector<char, boost::container::pmr::polymorphic_allocator<char> > >::ContainerRefObject<o2::framework::MessageContext, unsigned long> (index=0, bindingChannel="from_tpc-tracker_to_tpc-entropy-encoder", headerMsg=..., context=0x55c29c65d020, this=0x55c29ed9fe70) at /home/qon/alice/sw/SOURCES/O2/v1.2.0/0/Framework/Core/include/Framework/MessageContext.h:171
#15 o2::framework::MessageContext::VectorObject<char, o2::framework::MessageContext::ContainerRefObject<std::vector<char, boost::container::pmr::polymorphic_allocator<char> > > >::VectorObject<o2::framework::MessageContext*, std::unique_ptr<FairMQMessage, std::default_delete<FairMQMessage> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, unsigned long> (this=0x55c29ed9fe70) at /home/qon/alice/sw/SOURCES/O2/v1.2.0/0/Framework/Core/include/Framework/MessageContext.h:226
#16 std::make_unique<o2::framework::MessageContext::VectorObject<char, o2::framework::MessageContext::ContainerRefObject<std::vector<char, boost::container::pmr::polymorphic_allocator<char> > > >, o2::framework::MessageContext*, std::unique_ptr<FairMQMessage, std::default_delete<FairMQMessage> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, unsigned long> () at /usr/lib/gcc/x86_64-pc-linux-gnu/9.3.0/include/g++-v9/bits/unique_ptr.h:857
#17 o2::framework::MessageContext::make<o2::framework::MessageContext::VectorObject<char, o2::framework::MessageContext::ContainerRefObject<std::vector<char, boost::container::pmr::polymorphic_allocator<char> > > >, std::unique_ptr<FairMQMessage, std::default_delete<FairMQMessage> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, unsigned long> (this=0x55c29c65d020) at /home/qon/alice/sw/SOURCES/O2/v1.2.0/0/Framework/Core/include/Framework/MessageContext.h:384
#18 o2::framework::MessageContext::add<o2::framework::MessageContext::VectorObject<char, o2::framework::MessageContext::ContainerRefObject<std::vector<char, boost::container::pmr::polymorphic_allocator<char> > > >, std::unique_ptr<FairMQMessage, std::default_delete<FairMQMessage> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, unsigned long> (this=0x55c29c65d020) at /home/qon/alice/sw/SOURCES/O2/v1.2.0/0/Framework/Core/include/Framework/MessageContext.h:371
#19 o2::framework::DataAllocator::make<std::vector<char, std::allocator<char> >, unsigned long> (this=this@entry=0x55c29c65d1b8, spec=...) at /home/qon/alice/sw/SOURCES/O2/v1.2.0/0/Framework/Core/include/Framework/DataAllocator.h:110
#20 0x00007f7c9c699780 in o2::tpc::<lambda(o2::framework::InitContext&)>::<lambda(o2::framework::ProcessingContext&)>::operator()(o2::framework::ProcessingContext &) const (__closure=<optimized out>, pc=...) at /usr/lib/gcc/x86_64-pc-linux-gnu/9.3.0/include/g++-v9/bits/move.h:74

davidrohr · 2020-05-15T10:20:49Z

For reference, besides the above 2 issues, shared memory transport is fully working for the TPC workflow with TPC ZS Raw data, ClusterNative, and Compressed Clusters all moved via shared memory as zero-copy and the shared memory region is also registered correctly for GPU DMA transfer. (Needs this PR + #3528) Very nice! Let's get the above 2 issues solved and then LGTM!

rbx · 2020-05-15T10:21:11Z

When shmem transport tries to initialize a buffer and there is not enough memory, it assumes some messages are already "in flight" and will be freed eventually, so it just keeps trying until memory becomes free, or until device state changes. Admittedly this doesn't handle the case where you ask for more than the segment size. Or if you are creating multiple message before starting to send them out. I would be fine to switch the logic to throw a transport exception here, if you prefer. That way it's in your hands how to proceed if there is no memory.
This is an intentional race condition. We don't want to have a "master" device that is creating the shared memory segment. So the first device that requests shared memory will either find it or create it (this is properly locked). That's why you need to pass the segment size to every (shmem) device.

davidrohr · 2020-05-15T10:32:36Z

OK, that explains it.
For 1. : Could we on DPL level make sure to set the maximum shared memory size for all devices in case of a merged workflow?

For 2. : I think both approaches have there use cases, waiting or bailing out. So I would make it switchable. And then in my opinion the default inside DPL running on a laptop / desktop should be to throw an exception.

ktf · 2020-05-18T12:01:56Z

@davidrohr for 1. I suggest we track it in https://alice.its.cern.ch/jira/browse/O2-1434 and we merge this as is.

ktf · 2020-05-18T12:02:21Z

for 2. I agreed.

davidrohr · 2020-05-18T12:05:10Z

@ktf : This breaks the tpc workflow right now with default settings.

This reverts commit 312a569.

) This reverts commit 312a569.

rbx · 2020-05-20T17:46:02Z

FairMQ @ v1.4.17 is now available with the fix for 1. The default setting is to throw fair::mq::MessageBadAlloc when out of memory. Old behavior can be set by passing --shm-throw-bad-alloc false to a device.

@Barthelemy the changes from @ktf with the signal handling are also in the tag.
@ironMann @sy-c the optimization with the bulk region calbacks is also in the tag.

The tag also contains factory->CreateMessage(size, fair::mq::Alignment{64}) (or NewMessage(size, fair::mq::Alignment{64}) if you are in a device). However it is currently only implemented for shmem sender scenario. Other cases will ignore the argument for the moment. Their implementation is coming in the near future.

…p#3591)" (AliceO2Group#3609) This reverts commit 312a569.

DPL: default to transport=shmem on the same host

9dc94dd

ktf requested a review from a team as a code owner May 15, 2020 09:05

matthiasrichter approved these changes May 15, 2020

View reviewed changes

ktf merged commit 312a569 into AliceO2Group:dev May 18, 2020

ktf deleted the enable-shmem branch May 18, 2020 12:02

ktf added a commit that referenced this pull request May 18, 2020

Revert "DPL: default to transport=shmem on the same host (#3591)"

922b794

This reverts commit 312a569.

ktf mentioned this pull request May 18, 2020

Revert "DPL: default to transport=shmem on the same host" #3609

Merged

ktf added a commit that referenced this pull request May 18, 2020

Revert "DPL: default to transport=shmem on the same host (#3591)" (#3609

bb93f8a

) This reverts commit 312a569.

amorsch pushed a commit to amorsch/AliceO2 that referenced this pull request May 28, 2020

DPL: default to transport=shmem on the same host (AliceO2Group#3591)

75cbc3d

amorsch pushed a commit to amorsch/AliceO2 that referenced this pull request May 28, 2020

Revert "DPL: default to transport=shmem on the same host (AliceO2Grou…

f7b94ce

…p#3591)" (AliceO2Group#3609) This reverts commit 312a569.

Navigation Menu

Uh oh!

DPL: default to transport=shmem on the same host - #3591

DPL: default to transport=shmem on the same host#3591
ktf merged 1 commit into
AliceO2Group:devfrom
ktf:enable-shmem

ktf commented May 15, 2020

Uh oh!

ktf commented May 15, 2020

Uh oh!

davidrohr commented May 15, 2020

Uh oh!

ktf commented May 15, 2020

Uh oh!

ktf commented May 15, 2020

Uh oh!

davidrohr commented May 15, 2020

Uh oh!

davidrohr commented May 15, 2020

Uh oh!

davidrohr commented May 15, 2020

Uh oh!

rbx commented May 15, 2020 •

edited

Loading

Uh oh!

davidrohr commented May 15, 2020

Uh oh!

ktf commented May 18, 2020

Uh oh!

ktf commented May 18, 2020

Uh oh!

davidrohr commented May 18, 2020

Uh oh!

rbx commented May 20, 2020 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

ktf commented May 15, 2020

Uh oh!

ktf commented May 15, 2020

Uh oh!

davidrohr commented May 15, 2020

Uh oh!

ktf commented May 15, 2020

Uh oh!

ktf commented May 15, 2020

Uh oh!

davidrohr commented May 15, 2020

Uh oh!

davidrohr commented May 15, 2020

Uh oh!

davidrohr commented May 15, 2020

Uh oh!

rbx commented May 15, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

davidrohr commented May 15, 2020

Uh oh!

ktf commented May 18, 2020

Uh oh!

ktf commented May 18, 2020

Uh oh!

davidrohr commented May 18, 2020

Uh oh!

rbx commented May 20, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

4 participants

rbx commented May 15, 2020 •

edited

Loading

rbx commented May 20, 2020 •

edited

Loading